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ABSTRACT 


Technological advancements, especially in communications systems, 
have led to a burgeoning interest in network centric warfare (NCW), 
fundamentally changing how warfare is being conducted. Network centric warfare 
(NCW) systems are being rushed to the field and are offered as a solution for the 
‘fog of war’ and as a way to reduce manpower costs. To date, there are no 
empirical findings that support or refute these NCW system claims. 

The goal of this thesis was to ascertain the utility of the Geographical 
Recall and Analysis of Data in the Environment (GRADE) as a method and 
process by which complex human-technological systems can be assessed. The 
GRADE builds upon the Dynamic Model of Situated Cognition (DMSC). This 
study essentially determines if GRADE could be used in model validation in 
laboratory and field settings for evaluating NCW claims. Unfortunately, that 
research goal was not entirely realized due to constraints and limitations in the 
data collection exercise. 

The thesis discusses the ‘lessons learned’ from this research effort and 
makes recommendations about future exercises and how to better populate the 
DMSC with data. Additional recommendations for changes to the processes and 
procedures for data collection are provided. 
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I. INTRODUCTION 


A. PREFACE 

Accurate and timely information is critical to success in battle. Throughout 
history, commanders and foot soldiers have collected and transmitted battle 
information in various methods. These methods have included fire and smoke 
signals, homing pigeons, and runners. During WWII, new methods to exchange 
information included telephones, telegraph, and wireless radios. The twenty-first 
century has brought us fully into the digital age in which we transfer information 
to distributed locations over the globe by using wireless networks, the internet, 
and voice over IP (VOIP). 

These technological advancements in communication cause a change in 
how warfare is conducted. Network centric warfare (NOW) systems are being 
rushed to the field offered as a solution for the ‘fog of war’ and as a way to 
reduce manpower costs. Many of the new technologies that enable NOW have 
given little consideration to the role of the human in the design or implementation 
of the system. There is an unstated and untested assumption that system 
performance is facilitated by the introduction of any new piece of equipment. 
This thesis will describe a method to test these assumptions by objectively 
assessing system performance using a process tracing approach. This proposed 
method will provide NCW system developers with the ability to collect objective 
data on the total system’s contribution to a shared understanding of the 
battlefield and provide decision makers with information on the limitations and 
capabilities of this technology. 

This introductory chapter will first provide an overview of network centric 
warfare. Then a definition of Human System Integration (HSI) is provided to 
explain the process by which the individual domains of HSI are integrated in a 
human-centered design process. Lastly, three of the eight domains of HSI will be 
further defined within the context of NCW for the purpose of this thesis. 
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B. NETWORK CENTRIC WARFARE 

Network centric warfare (NCW) is based on a concept of operations in 
which humans use technology to share information and the ability to rapidly 
utilize that information will result in increased military effectiveness. The goal of 
NCW is to increase combat power by utilizing all assets in the field and at home 
to electronically link decision-makers and shooters. With access to the same 
information at the same time, NCW theory suggests that this new technology will 
enable operators to achieve shared awareness resulting in increased speed of 
command, higher tempo of operations, greater lethality, increased survivability 
and better synchronization (Alberts & Garstka, & Stein, 1999; Alberts & Hayes, 
2003; Cebrowski & Garstka, 1998). 

The Department of Defense (DoD) is currently developing and fielding 
numerous versions of this networked technology, but the goals of NCW cannot 
be achieved with technology alone. NCW is an emerging theory of war and a 
new concept of operations, and it provides an opportunity to develop a human- 
centered system of systems. Network centric warfare requires a cultural change 
away from traditional hierarchical military relationships to include the new ability 
of networking over the Internet among large groups of people to accomplish a 
combat objective in real time. The technology itself will not be an optimal force 
multiplier unless humans are recognized as critical information nodes and 
sensors within the system. 

In NCW the humans are vital to the system because they are required to 
define, forage and harvest the information needed to accomplish an objective. 
The goal then becomes developing a system that is both transparent and intuitive 
to the operator while providing critical information to decision makers in a timely 
manner. To this end, if a human-centered design is used in the development 
process, the cost for the entire life cycle of the NCW system may be reduced, 
allowing the overall system to operate more effectively. 


2 



C. HUMAN SYSTEMS INTEGRATION 

Human Systems Integration (HSI) is a human-centered approach to the 
design, development, and acquisition of systems from cradle to grave. A 
fundamental premise of HSI is that any new materiel solution must consider the 
humans involved in the system. The acquisition process should include human 
considerations early in the design, thereby enhancing human-system 
performance. Implementing HSI early in the acquisition process can result in a 
reduction of total lifecycle costs, and better overall system performance. 

To achieve a human-centered design, HSI professionals provide a 
comprehensive management and technical strategy for human systems 
integration. This HSI plan is a DoD requirement as stated in the 5000.2 series 
instruction document (USD/AT&L, 2002). While there is considerable debate 
about the number and names of the HSI domains, the Naval Postgraduate 
School, which offers the only graduate degree in HSI, uses eight domains in its 
definition of HSI. These eight domains are: Human Factors Engineering, 
Manpower, Personnel, Training, Human Survivability, Health Hazards, System 
Safety, and Habitability. Following is a description of each of these eight HSI 
domains. 

1. Training 

The instruction, education, and training required to provide personnel with 
the knowledge, skills and abilities needed to operate and maintain systems 
(NPS, 2006). For example, on-the-job training is an organization’s attempt to 
assist task specific learning through instruction, observation, or practice that 
reflects the demands of a job. 

2. Personnel 

The knowledge, skills, abilities, aptitudes, and experience required for the 
human component of the system to enable the system to achieve full capability 
(NPS, 2006). Personnel selection is based upon the tasks that are to be 
performed to complete a job. For example, operators, maintainers, and support 
personnel would have different KSAs and experience. 
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3. Human Factors Engineering 

A design and development process that focuses on the interaction 
between human beings and the tools, equipment, machines and systems they 
employ to achieve system effectiveness and safety (NPS, 2006). There are two 
objectives within a human-centered design: consider ergonomic design (the 
hardware) and create an intuitive and transparent system (software) for the 
operator. 

4. Manpower 

The total demand, expressed by the number of individuals, required to fill 
jobs, slots, or billets for a system with consideration for total life cycle 
requirements to include the big areas of both operational and support (NPS, 
2006). For example, manpower would represent the number of bodies required 
to supervise, train, maintain and operate an aircraft, but does not represent KSAs 
required for each billet. 

5. System Safety 

System safety is the application of engineering, management, and HSI 
principles, as well as human factors design criteria, and techniques to achieve 
acceptable levels of risk hazard for mission requirements and system capability 
(NPS, 2006). For example, a weapon system is designed to reduce the chance 
of death or injury to operators within the constraints of operational effectiveness, 
cost, schedule, and performance throughout all phases of the system life cycle. 

6. Human Survivability 

Fluman survivability is the ability of a system and its personnel to avoid or 
withstand hostile environments without jeopardizing mission accomplishment 
(NPS, 2006). For example, in the case of a tank, it is the ability of the crew and 
the overall system to continue to function during and after a disturbance from 
something as simple as a rollover due to terrain or as complex as withstanding a 
direct hit from an armor piercing round. 

7. Health Hazards 

Health hazards are anything that can have a harmful effect on health 
either during the system’s production, operation, or maintenance to include 
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decreased worker performance and significant long-term health risks (NPS, 
2006). For example, health hazard specialists may be concerned with the 
toxicity/lethality of jet fuel and the personal protection equipment (PPE) required 
for personnel to work in and repair aircraft fuel cells. 

8. Habitability 

Habitability is concerned with the qualities of the physical living 
environment and support services within a system which lead to mission 
effectiveness (NPS, 2006). Using a space station as an example, it is important 
that the crew resting quarters are away from heat and noise (physical stressors), 
the work area has adequate space to prevent overcrowding (psychological 
stressors), and that there are adequate facilities for food preparation, bathing and 
waste removal. These accommodations will contribute greatly to the 
performance of the crew and the total system. 


D. THE RELATIONSHIP BETWEEN NOW AND HSI 

This thesis uses a human-centered approach that encompasses the 
following three domains of HSI: training, personnel, and human factors 
engineering. These three domains will now be defined within the context of 
NCW. 

1. Training 

NCW systems require a transition from legacy systems and concepts of 
operations to information age weapons, equipment, and technology (Money, 
2001). This transition also requires new concepts of operations that facilitate 
new strategies and tactics by providing training for decentralized rapid operations 
for use in asymmetrical warfare. NCW systems require that operators receive 
training in both computer/digital skills and in communication and coordination 
within horizontal and vertical networks. 

2. Personnel 

To reduce training time and cost for NCW systems, it is imperative to 
select and place the correct personnel in the right jobs (Money, 2001). For the 
NCW system to perform optimally, personnel will need to have information 
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technology skills, an aptitude for multitasking, and experience in a position that 
requires the individual to fulfill many roles and duties simultaneously. 

3. Human Factors Engineering 

NCW systems need to have a human-centered design to maximize their 
potential (Money, 2001). For example, designing computer displays that support 
situational awareness and control of real-time and dynamic processes for NCW 
means more than just presenting the data in a form that supports the workflow of 
the operator. To enable effective understanding of the battlespace, it is 
necessary to portray the information in a manner that supports the cognitive 
capability and goals of the operator. HFE is critical in all cases where the 
operator is performing at the extremes of his or her performance envelopes. 
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II. LITERATURE REVIEW 


This chapter reviews literature from the fields of network centric warfare 
(NCW), team performance, development of virtual teams, and situated cognition. 

A. NETWORK CENTRIC WARFARE 

The current direction of the DoD has shifted from platform-centric warfare 
to NCW or network centric operations (NCO) (Cebrowski & Garstka, 1998). The 
potential of information age warfare was first recognized in the 1983 invasion of 
Grenada, reinforced in the 1990 Persian Gulf War, and widely accepted in the 
2003 invasion of Iraq and the war on terror in Afghanistan (Wilson, 2004). 
Networked information technologies supplied the forces with unprecedented 
battlefield advantages during these wars. These information technologies are 
capable of providing leadership with real-time awareness of current theaters. 
Reportedly, operators in the field who use NCW systems are also experiencing 
increased situational awareness on the battlefield (Garstka, 2003; Blash, 2003). 
The current Department of Defense trend is to reshape the US military as a 
network centric force that will be able to leverage information to increase speed 
and combat effectiveness. 

NCW combines various types of information technologies to produce 
information superiority that is reputed to increase combat power through self¬ 
synchronization and other network centric operations (Alberts, 1999; Alberts & 
Garstka, 2001; Wilson 2004). A robust, secure and broadband network plays a 
crucial role in future warfare in which the network will connect all types of sensors 
in the air, on land, and under the sea. Ideally, the network will assist with 
logistics to ensure the timely supply of troops and help synchronize conventional, 
electronic, and information attacks. 

B. TEAM PERFORMANCE 

Outcomes in time-critical, high-risk situations are often dependent on 
crisis action teams or “tiger teams” in which expertise, information, and tasks are 

7 



distributed across highly trained and specialized individuals. The effectiveness of 
these teams requires rapid, complex, coordinated behavior, resulting in 
integrated team performance. The high degree of interdependence that is 
inherent in such teams makes the performance requirements of individual 
members particularly demanding (Kozlowski, Gully, McHugh, Salas, Cannon- 
Bowers, 1996a). Previous research indicates that the contribution of team 
members is not simply an additive function of each individual. Rather, the 
patterns of communication, information flow, and responses which indicate a 
shared mental model of team members enables the team to respond to 
externally driven and constantly changing task demands (Thordsen & Klein, 
1989; Fleishman & Zaccaro, 1992; Weick & Roberts, 1993). To achieve high 
team effectiveness, it is believed that teamwork-building or enabling skills may be 
needed to facilitate a congruent and synchronous mental model of action for 
team members (Kleinman & Serfaty, 1989). 

Thus far, research in the field of team performance has led to an 
increasing focus on work teams within organizations and on how best to build 
teamwork skills that are the foundation of team effectiveness (Kozlowski, Gully, 
McHugh, Salas, Cannon-Bowers, 1996b). The literature suggests that any team, 
including a command and control team, requires a certain amount of time to 
become effective. However, the opportunity to spend time together as a team is 
not always available. This can be seen in real world emergencies where 
effective ad-hoc teams come together in a short period of time. For example, 
teams formed quickly for Hurricane Katrina emergency care. And in NCW a 
team may be formed at any time from members distributed around the globe 
(Kayes, 2003). 

In the scientific literature, teams are distinguished from other collectives by 
the characteristic of shared interdependent work (Thordsen & Klein, 1989; Salas, 
Dickinson, Converse, & Tannenbaum, 1992). In comparison, terms “group” or 
“work group” include work collectives who see themselves and are seen by 
others as social entities with common goals but with looser task connections 
(Thordsen & Klein, 1989). 
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Several theories describe the stages teams pass through as they evolve 
(Bennis & Nanus, 1985; Bennis & Shepard, 1956; Bion, 1961; Schultz, 1958; 
Tuckman, 1965; Tuckman & Jenson, 1977). These models provide a description 
of developmental stages of teams and suggest that team development is a 
process that occurs over time. The models further indicate that much of the time 
required to develop the team is spent on social rules and norms that reduce 
interpersonal conflict while increasing the productivity of the team. For example, 
Tuckman’s (1965) model consists of the following four stages: forming, storming, 
norming, and performing. Although this model is a stage model, it does not imply 
a progressive directionality to a team’s ability to pass through the stages. In 
other words, a team can revisit any of the stages, for example, going from 
forming to storming and then back to forming again. These, stage models 
suggest a time-consuming process to develop an effective team. Although the 
issues mentioned are one aspect of team development, the models are not able 
to explain rapid ad-hoc team formation during time-critical events such as military 
operations. 

C. VIRTUAL TEAMS 

Understanding team performance becomes even more important for NCW 
because technological advances, globalization, organizations, and global 
militaries are moving toward utilizing more network structures and team-based 
functions (Lipnack & Stamps, 1997). The term “virtual team” is becoming more 
prevalent as teams move from being primarily “co-located,” where team 
members are in one physical location, to “virtual,” where team members may be 
located anywhere on the globe, separated by space and time. 

For definition purposes, virtual teams will be considered groups of 

individuals who interact through various communication technologies to 

accomplish common goals. Virtual learning teams are being used in education 

and corporate training programs in an attempt to enhance collaboration and 

cooperative learning experiences. Even though various studies of groups using 

computer-mediated communication have contributed to an increased 
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understanding of both face-to-face and virtual teams, the results are inconclusive 
(McGrath & Hollingshead, 1994). Most studies comparing face-to-face groups 
and groups using communication technology suggest that face-to-face teams are 
more effective. McGrath and Hollingshead (1994) examined fifty studies on 
computer-assisted group performance and found that computer-mediated groups 
tend to have fewer interactions and less information exchange among members 
than face-to-face groups. Virtual team members can exchange verbal information 
as efficiently as a face-to-face team, but they lack the ability to convey non-verbal 
cues as easily as individuals that are co-located. This lack of non-verbal 
communication can contribute to increased misunderstanding among members 
(Warkentin, Sayeed, & Hightower, 1997). Research also indicates that face-to- 
face teams have better internal leadership and coordination than virtual teams 
(Burke & Chidambaram,1994; Eveland & Bikon, 1989). 

Theoretical development and empirical research are needed to better 
understand and respond to the challenges that virtual teams face (Furst, 
Blackburn, & Rosen, 1999). Although there have been several related studies in 
this area, few research efforts have focused on rapid-forming, ad-hoc successful 
teams. In the case of real world examples, which are often investigated after the 
fact, these temporary, newly-formed team members have a limited history of 
working together and have few prospects of working together in the future. 
Therefore, the question exists: how is it that they succeed? 


D. DISTRIBUTED SOCIAL NETWORKS 

Many organizations are similar to networked communities in that they 
have multiple sets of work team members (including multiple supervisors), 
geographically separated relationships, and teams of co-workers shifting by the 
day and week as employees get involved in multiple projects. The situation is 
different from that dealt with by traditional organizational theory, which defines 
organizations as densely knit workgroups neatly structured in bureaucratic, 
hierarchical organizational trees (Wellman, 1997). How do people work together 

in large, sprawling, networked organizations where they are members of multiple, 
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transitory, physically dispersed teams simultaneously? In particular, how do 
people in such organizations obtain knowledge from others when they do not 
know whom to ask? 

These questions are of immediate practical importance for NCW. The key 
to making NCW effective is not only in connecting geographically dispersed 
forces, retrieving information from sensors, human intelligent agents and other 
information sources, but also analyzing different information, generating useful 
knowledge and then distributing it to the right person at the right time in the right 
format. This enormous task explains why computer-supported solutions are 
being developed to work through trusted interpersonal relationships to identify, 
locate, and receive information within and between communities and 
organizations (Wellman, 1997). It is not surprising that research in knowledge 
management (KM) has been driven by computer scientists and communication 
scientists interested in building tools for harvesting knowledge and then 
managing the growing database. The critical issue of tracking who knows what 
information at any given time is a more complex task in networked organizations 
(Cross & Borgatti, 2000). Traditionally in an office setting, an individual will first 
examine documentation or other help sources from their desks before wandering 
out into a hallway in search of friendly colleagues. The problem becomes acute, 
however, in distributed communities (Ackerman & McDonald, 1996). 


E. SITUATIONAL AWARENESS 

Situational awareness (SA) is a concept that grew from lessons learned in 
the battlefield. SA has been defined as “adaptive, externally directed 
consciousness” (Dzierzanowski, 1999). The United States Army has defined it 
as the ability to maintain a constant, clear mental picture of the tactical situation 
(Dzierzanowski,1999). This mental picture includes an understanding of the 
terrain and the relationship between friendly and enemy forces 
(Dzierzanowski,1999). 
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SA as defined by Endsley as: “...the perception of the elements in the 
environment within a volume of time and space, the comprehension of their 
meaning, and the projection of their status in the near future” (Endsley, 1995; 
Wickens et al., 2001). 

The definition by Endsley, as applied to an object in the environment tries 
to answer these questions: what is it, what is it doing, why is it doing it, and what 
will it do next. However, this definition misses an important element of SA which 
is the context within one’s own situation and not just an awareness of external 
objects. SA should also include an acknowledgement of one’s own 
psychological state (Omedei, Wearing, McLennan & Hansen, 2001). Not all 
objects in the environment need to be monitored all the time. At any given time 
some elements will be less important than others, but Endlsey suggests that 
some level of awareness is required of each element, to monitor changes in 
levels of importance (Endsley, 1993). 

Another view of SA is that it is a detailed appreciation of what is 
happening now and what is likely to happen in the immediate future. This 
statement can be compared with commander’s intent which focuses on the 
overall end state to be eventually achieved from an understood start state. 

According to Endsley, an operators’ SA exists in one of the following three 
levels of complexity (1995, 2001): 

Level 1 consists of perception of elements in the environment. 

Level 2 involves comprehension of the current situation. 

Level 3 incorporates the prediction of the future actions of data elements. 

Considerable effort has been invested in devising methods of quantifying 
SA, measuring SA and training to improve the ability to acquire SA (Gawron, 
2000; Nofi, 2000; Endsley, 2001; Endsley & Robertson, 2000). Methods of 
measuring SA range from the highly intrusive to the deliberately unobtrusive. In 
the former, the activity is suddenly stopped and the immediate SA of participants 
is captured in some way usually a series of questions to be answered for later 
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comparisons with ground truth at that time. One issue with this method is that it 
interrupts the activity, possibly degrading the level of immersion the participant 
has in the exercise/experiment. Unobtrusive measures typically require 
participants to self rate their SA retrospectively. This post hoc reporting 
procedure relies on participants’ memories of events that may have occurred 
much earlier in time. SA at any point in time is unlikely to be fixed in long term 
memory because SA is a process that ebbs and flows with ground truth. A 
retrospective query to asses SA may provide an average SA score over the 
entire scenario or a specific score for a single point in time. 


F. SITUATED COGNITION 

NCW theory makes a fundamental assumption that improved information 
infrastructures will facilitate military decision-making and, therefore, military 
effectiveness. Previous work linking NCW applications to military effectiveness 
efforts have had difficulties in modeling the decision-making aspects of the 
process (Hazen et al., 2003). Advocates of NCW suggest the prime advantage of 
NCW systems is in the generation, communication, and mining of operational 
information. 

To test these premises, it is necessary to empirically assess the 
contribution of any new technology and other contributions to the NCW system 
such as training and personnel selection. Therefore, a model which assesses 
the effectiveness of an NCW system must include a decision-making model. 
Models of information throughput and human decision-making should inform 
decision makers of the capability of weapon systems. These models should not 
assume that all information is accurate, understandable, timely, cognitively 
matched, and fully used by the decision-maker. In fact, previous research 
indicates that it is not the quantity or quality of the data but rather the contextual 
nature of the information that is important in determining value to a decision¬ 
maker (Fewell & Hazen, 2004). 
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The Dynamic Model of Situated Cognition (DMSC) provides a conceptual 
model that is useful for modeling military decision making (Miller & Shattuck, 
2004; Miller & Shattuck, 2005b). The schema used for classifying decisions is 
based on Naturalistic Decision Making (NDM) theory (Klein, 1997; Shattuck & 
Miller, 2005). In this approach, process tracing provides a robust understanding 
of contributions of both the operators and the system to a decision-maker’s 
actions (Miller & Shattuck, 2004; Miller & Shattuck, 2005a; Shattuck & Miller, 
2004). The DMSC is composed of 6 ovals and three lenses (see Figure 1). 



All data in the 
environment 


Projection of 
decision maker 

© Miller and Shattuck, 2003 


Figure 1. Dynamic Model of Situated Cognition (From: Miller & Shattuck, 

2004). 

Oval 1 consists of all data in the battle space to include blue and red 
forces, non-combatants, terrain features, weather, and sensor location. Oval 2 
represents the data collected by the sensors. Oval 2 is smaller than Oval 1 due 
to the fact that no sensor array will detect everything in the environment. Oval 3 
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represents data displayed on operator screens. Oval 4 is the operator’s 
perception of the data. Oval 5 represents the comprehension while Oval 6 is a 
projection of current data. 


The model also has 3 lenses (labeled A, B, C in Figure 1) which mediates 
how information is processed by the decision maker. According to the model, the 
lenses direct the attention of the decision maker toward certain data, and in some 
cases, skew what information is perceived (See Figure 2). Thus, as lenses 
change shape, attention will shift; decisions may be influenced by the change in 
these lenses (Miller & Shattuck, 2006). 




© Miller and Shattuck, 


Figure 2. Distortions in the lenses lead to inaccurate perceptions, 
comprehensions, and projections (From: Miller & Shattuck, 2006). 


Miller and Shattuck suggest that each lens has a purpose which is listed in 
Table 1 and that there are currently 6 classes of information within the lenses as 
can be seen in Table 2 (Miller & Shattuck, 2005). 
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Table 1. Purpose of Each Lens. 


Lens 

Proposed Purpose 

Lens A 

Directs attention 

Lens B 

Mediates information organization 

Lens C 

Course of action 


Table 2. Classes of Information Embedded in Each Lens. 


• Individual states and traits 

• Social factors 

• Local context 

• Plan 

• Guidelines 

• Experience _ 


Enhancements to the original model include the addition of quantifying 
accuracy and certainty, feedback loops (see Figure 3) that represent the iterative 
nature of decision-making, and to teams (see Figure 4) (Miller & Shattuck, 2006). 



Figure 3. Feedback Loops in the DMSC (From: Miller & Shattuck, 2005). 
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In the case of NCW it is important to have a model that incorporates not only 
each technological system input, but also the various inputs from each human 
node in the system. NCW, by its very nature, is considered to be a collaborative 
effort and as can be seen in Figure 4, the DMSC is capable of representing these 
team interactions. 



Figure 4. DMSC Applied to Teams (From: Miller & Shattuck, 2005). 

The merit of the DMSC model has been recognized by multiple 
organizations. Flowever, to fully test the model and explore its use as a method 
to evaluate complex systems, there needs to be a reliable and accurate method 
for determining the contents of each of the six Ovals of the DMSC. One 
candidate tool for accomplishing this is a situated cognition measurement tool 
called the Geographical Recall and Analysis of Data in the Environment 
(GRADE). GRADE was conceived as a measure to illuminate what participants’ 
were attending to during a Future Combat Systems experiment (Miller & 
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Shattuck, 2006). Currently GRADE provides only qualitative data to the 
researcher, but that data provides situated cognition over time of what each 
participant knew of the battlespace and when they knew it. GRADE has 
continued to demonstrate efficacy in populating Oval 4 in multiple field studies 
(Miller & Shattuck, 2003; Miller & Shattuck, 2005; Miller & Shattuck 2006). 

This thesis describes using GRADE as method and process by which the 
DMSC model can be validated and applied in the field setting for NCW research. 
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III. METHODS 


A. OVERVIEW 

As previously stated, there is not a well-defined methodology for 
evaluating the human and technological components of a network system. This 
shortcoming in NCW makes it difficult to determine whether hardware, software, 
or changes to organizational structures or procedures really improve warfighting 
capabilities. This thesis uses the DMSC as the theoretical framework for NCW 
and, in addition, proposes GRADE an evaluation strategy that is based on that 
model. GRADE examines the contents of each of the six ovals in the model to 
determine who knew what and when they knew it. This methodology also 
nominates appropriate metrics and discusses techniques for analyzing data in 
both laboratory and field settings. 

While GRADE has been used in several laboratory and field experiments 
to assess command and control activities (Miller & Shattuck, 2007), the tool itself 
has not been assessed to determine its efficacy in such data collection venues. 
GRADE provides a method to collect a participant’s situated cognition and then 
use that data to populate Ovals 4 and 5 of the DMSC. With Ovals one through 
five populated it is then possible to trace how data and information propagate 
through the NCW system over time. 


B. VENUE 

The Naval Postgraduate School conducts field exercises quarterly in the 
Tactical Network Topology (TNT) program. This series of exercises provides an 
opportunity to collect NCW data in a field environment. Since the program is 
receptive to input from investigators in the design and development of scenarios, 
it is possible to influence data collection activities. The Naval Postgraduate 
School (NPS) TNT Program began approximately four years ago with the 
purpose of providing the opportunity for students and faculty to evaluate some of 
the latest technologies in an operational environment and, when appropriate, to 
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rapidly transition these technologies to the warfighter. The program relies heavily 
on the operational knowledge of the NPS faculty and student body as well as a 
very close working relationship with United States Special Operations Command 
(USSOCOM). Congressional funding from the Center for Defense Technology 
and Education for the Military Services (CDTEMS) together with funding from 
USSOCOM and the Office of Force Transformation (OFT) has permitted one to 
two week long quarterly field exercises to be conducted, using laboratories on 
the NPS campus, the NPS Beach Laboratory and the Center for Interdisciplinary 
Remotely-Piloted Aircraft Studies (CIRPAS) (Marina Airport), Monterey Bay, the 
Military Operations on Urban Terrain (MOUT) facility at Fort Ord, U.S. Coast 
Guard (Oakland) facilities, and an NPS CIRPAS UAV test facility at the California 
Army National Guard, Camp Roberts, CA. 

The field exercises began with a series of Surveillance and Target 
Acquisition Network (STAN) exercises which have transitioned to the Tactical 
Network Topology (TNT) Program which focuses on providing the soldier on the 
ground the ability to push or pull video, data, text messages, and use voice over 
IP (VOIP) to communicate with other soldiers and with other ISR assets in both 
rural and urban environments. In addition the network provides screens that 
display ‘situational awareness’ and blue force tracking. It utilizes air and ground 
based wireless networks, satellites, unmanned vehicles (UAVs, UGVs, AUVs, 
airships, tethered balloons), unattended ground sensors, handheld PDAs, and 
laptops to enhance situational awareness and to enhance our ability to find, fix, 
and identify enemy personnel and equipment. 

TNT 07-03 was conducted 11 through 18 May 2007 at Camp Roberts, 
California. This field exercise consisted of the following scenario activities: 
Perimeter Security, Border Surveillance, Area Search, and Identify/Monitor 
Area/Activity. The data for this research effort were collected on the morning of 
17 May 2007 during a baseline scenario. 

The assigned mission for the five members of the Blue Forces was to 
monitor the borders for trespassers. In the event a trespasser was spotted, they 

were to conduct surveillance and monitor their activity. 
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The assigned mission for the three members of the Red Force was to 
cross the borders at different locations. Red 1 was to drop off the material for an 
lED at a certain location. Later Red 2 would stop by the predetermined location 
and assemble the lED. The last part of the mission was to have Red 3 set up the 
lED at that location for use and simulate detonating the device. 

1. Participants 

Data were collected on participants in the following positions during the 17 
May 2007 morning scenario. Participants in the following positions completed 
the GRADE measures and wore digital audio recorders. 

• Tactical Operations Center (TOC) Commander 

• Air Boss 

• Raven 3 (UAV) 

• Buster (UAV) 

• SORSE (five individuals) 

• Red force commander 

• Red vehicle 1 

• Red vehicle 2 

• Red vehicle 3 

Although Raven 3 participated in the exercise, the GPS location data on 
this UAV were never captured due to technical difficulties with Cursor on Target 
software making it impossible to establish ground truth for this vehicle. The 
Raven 3 GRADE data were compared to the other GRADE data to determine if 
participants correctly populated the measures. 

Special Operations Research, development, test & evaluation Support 
Element (SORSE) is the Blue force ground troops and was composed of five 
individuals. Three of them rode on ATVs while the other two drove in SUV’s. 
Redi, Red2, and Red3 comprised the red force ground troops. All three drove 
their own civilian vehicles. 
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2. Apparatus and Instrumentation 

As discussed previously the DMSC is composed of 6 ovals. The first 
three ovals are comprised of the technological system’s contribution and the last 
three ovals are the human contribution to the total system. It follows that 
measurements should follow this division. Table 3 indicates the measures that 
could be used to populate these ovals. In the case of the current research 
project, data were captured primarily for Ovals 1, 2, 3, 4 and 5 of the DMSC for 
the TNT 07-03 field experiments. 


Table 3. Measures used for populating DMSC. 


Oval 

TNT Field Experiment 

1 

GPS network 

Network data storage 

2 

Sensor network 

Network data storage 

Sensor capabilities 

3 

Screen captures for individual 
players 

4, 5,6 

GRADE 

Digital voice recorders 

Acetate overlays (Oval 6) 


To maintain a consistent temporal reference, both the GRADE measures 
and digital voice recorders were synchronized to the GPS network time. To 
capture data for Oval 1, ground truth data were collected from each vehicle’s 
GPS tracking system and stored by the TNT network. Commercial GPS units 
were used by the ground forces; UAV position was captured by software 
broadcast of GPS position. In the case of the ground troops, the GPS units were 
linked with laptop computers that had wireless connectivity with the TNT network. 
GPS data were sampled at a rate of once per second for the duration of the 
scenario. Figures 5 and 6 provide examples of GPS data collection and storage. 
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Figure 5. Tech Data. 



Figure 6. Tech Data + time. 
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Oval 2 data were captured first by understanding a sensors’ capability and 
secondly by providing data on where the sensor was looking. In the case of the 
former, a full understanding of what a sensor could see or hear is taken into 
consideration for correctly identifying hits and misses of participants. For 
example, an audio sensor is not going to provide a picture enabling participants 
to visually ID a vehicle; similarly, a small UAV flying at 10,000 feet is not going to 
have the resolution required to allow identification of moving ground targets. In 
the latter case, it is not enough to know where a sensor is, but it is also essential 
to know where it is looking. The Scan Eagle UAV was capable of providing this 
capability by providing its own location and the location where the camera was 
looking. This was achieved by software that could calculate the location the 
camera was looking by calculating camera position versus UAV location. With a 
field of view calculation researchers could theoretically determine the exact area 
viewed by the sensor throughout an exercise. Unfortunately, the Scan Eagle 
UAV did not participate in the 17 May AM scenario. 

Oval 3 was populated by collecting screen captures of what was on 
participants’ displays at fifteen minute intervals (see Figures 5 and 6). 

Oval 4 and 5 data was captured by a combination of both the situated 
cognition measure GRADE and digital voice recorders. GRADE was collected in 
this case using a map of the area of operation for the TNT 07-03 experiments 
(see Figure 7). Multiple blank GRADES, a clipboard, and a blue and red Sharpie 
fine point permanent marker were provided to each participant. 
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Figure 7. GRADE example. 

Nine Olympus WS-300M digital voice recorders were used to collect audio 
data from the participants. To ensure the maximum understanding of voice data, 
omnidirectional microphones were used in conjunction with the digital voice 
recorders. 

3. Procedure 

The procedure section is organized to indicate how data was collected for 
each Oval of the DMSC during the TNT 07-03 field experiment. This section will 
start with Oval one and conclude with Oval five. 

a. Oval 1 

Careful advance collaboration with TNT exercise planners enabled 
an understanding of what sources of data were available to be collected and then 
used to populate the first three Ovals. For Oval 1 all UAVs and ground vehicles 
were equipped with GPS units that were able to broadcast to the TNT network. 
Location data was sampled and saved once per second. 

b. Oval 2 

For Oval 2 all data provided by technological sensors were time 
stamped as the information was passed into the TNT network. 

c. Oval 3 

For Oval 3 a screen capture of what was presented on the displays 

was captured every 15 minutes. In addition, in the Tactical Operations Center a 
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time-stamped digital log was kept of what UAV video feed observers noted. 
Finally, the TOC COM and Air Boss wore digital voice recorders providing further 
information on what information was available on the displays. 

d. Oval 4 and 5 

The decision makers listed in the Participants section (above) were 
trained in the GRADE protocol. The GRADE was administered every twenty 
minutes during the exercise. The Air Boss would announce over the radio 
network time to complete a GRADE to the Blue forces while Red Commander 
would make the same announcement to Red forces. Participants were instructed 
to provide their understanding of the battle space on the GRADE maps (see 
appendix A). The GRADE instructions were as follows: 

Please complete the GRADE every 20 minutes after STARTEX 
until FINEX according to the following instructions. 

1. Turn away from your information display, monitor etc. 

2. Record your duty positions (e.g.. Air Boss, Red1, etc.) and the 
current time in the upper left portion of the map. 

3. Use the blue marker to identify where you believe the blue 
forces are located. Draw a circle around each blue entity to 
indicated your confidence/certainty. A small circle Indicated 
high confidence/certainty. A large circle indicates low 
confidence/certainty. 

4. Use the red marker to Identify where you believe the red 
forces are located. Draw a circle around each red entity to 
indicate your confidence/certainty. A small circle Indicates 
high confidence/certainty. A large circle Indicates low 
confidence/certainty. 

The top left corner of the GRADE map indicated the TNT 
experiment and date of data collection. Blanks were provided for participant 
position information and time the GRADE was conducted. 
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4. Data Analysis Plan 

Prior to collecting the data at Camp Roberts, a plan for analyzing the 
GRADES was developed. The plan included converting latitude/longitude data 
provided by GPS devices to military grid coordinates. Then, a master location 
spreadsheet would be created that listed the military grid coordinates for all 
relevant entities in the battlespace for each time the GRADE was administered. 
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Figure 8. Location data spreadsheet. 


Military grid coordinates for each participant’s entries on the GRADE sheets 
would also be determined. The participants’ marks for each entity would be 
compared with the actual (ground truth) location and the distance between the 
actual (ground truth) and the recorded (actual) would be calculated (see Figure 8 
for an example). 

With the DMSC populated with data it is now possible to use the GRADE 
data (Ovals 4 and 5) to look back through Ovals 1, 2, and 3 to determine the 
errors, if there were any, in a participants’ perceived ground truth of the 
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battlespace. For example, a researcher may notice a point where a participant 
incorrectly identified a vehicle. The data trace would possibly provide this 
information: a Red force unit crosses a border (Oval 1), but is not yet detected. A 
UAV (sensor) flies by the enemy capturing an image (Oval 2). The video feed 
flows through the network and is displayed on an observers screen (Oval 3). The 
participant notices the vehicle (Oval 4) and then incorrectly labels it a Blue force 
vehicle (Oval 5). Now the researcher can try to determine where the error was 
introduced. Was it because the display did not provide enough resolution (Oval 
3) or due to the lens between Ovals 4 and 5 indicating to the participant only Blue 
forces are in that area. 


Figure 9 illustrates the GRADE measures ability to capture a participant’s 
situated cognition (Ovals 4 and 5) over time. These figures include the ground 
truth overlays. 



SORSEGRADE3 SORSE GRADE 4+20 SORSE GRADE 5+20 

Figure 9. GRADE data and ground truth from SORSE at approximate 20 

minute intervals. 
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IV. RESULTS 


Due to the high rate of missing data, quantitative statistical analysis could 
not be completed for this thesis. Instead, a descriptive analysis of Participants’ 
performance on GRADE, ground truth vs. perceived truth, and a qualitative 
analysis of the digital voice recordings is provided. 

A. GRADE DATA 

The data collected from GRADE immediately indicated to the researchers 
that there are issues in how the participants populated the GRADE over the 
course of the scenario. As table XX indicates, nine participants completed 57 
GRADE measures over the course of the two hour experiment. Out of the 57 
GRADES generated, not a single GRADE was completed correctly indicating that 
participants did not follow instructions. To salvage the data collection, it was 
determined that if at least one Red or Blue force was clearly labeled, that 
GRADE would be considered usable. With this new criterion, 21 GRADE 
measures provided useful data indicating a 37% ratio for completed GRADES 
(see Table 4). 


29 



Table 4. Total GRADES vs usable GRADES. 


Position 

GRADES Compieted 

Usabie GRADES 

Air Boss 

7 

0 

Buster (UAV) 

7 

2 

Raven (UAV) 

6 

0 

Red Queen 

6 

0 

Red 1 

6 

5 

Red 2 

6 

6 

Red 3 

6 

4 

SORSE 

6 

4 

TOC CC 

7 

0 

TOTAL 

57 

21 


Further analysis of the 21 usable GRADES indicates only 44 usable data 
points out of a possible 282 data points resulting in 16% usable data for this 
research effort. This result is artificially low due to the assumption that 
participants would know where all players were at all times. A more accurate 
measure of GRADE performance is achieved by contrasting the 44 usable data 
points against the 131 actual data points found on the GRADE maps. In this 
case the GRADE measure provided 34% usable data return. 

As Table 5 indicates, there were a possible 513 total detections during the 
course of this scenario. Participants only indicated 262 actual detections yielding 
a 51% success rate of detecting either red or blue forces. Both Blue (105) and 
Red (50) forces did better at detecting themselves than at detecting the opposing 
force. 

Blue force had a total of 167 detections indicated on the GRADE 
measures. 62 of those detections were of Red forces while the remaining 105 
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were of friendlies indicating a 37% rate of detecting enemy forces. Red force 
had fewer players in the scenario, so it is not surprising that their total detection 
of 95 is lower than the Blue force. Red force had 45 Blue force detections 
indicating a 47% rate of enemy force detections. 


Table 5. Detections. 


Detections 

Blue 

Red 

Total 

Possible 

342 

171 

513 

Actual 

150 

112 

262 

By Blue 

105 

62 

167 

By Red 

45 

50 

95 


Table 6 indicates a different result for the TOC Commander who has the 
potential capability of having positions for all 6 Blue force assets and 3 Red force 
assets. The same measurements already described were used to populate this 
table. 


Table 6. Qualitative examination of TOC CDR GRADE data. 


Ground force 

Know Self/friendly 

Enemy 

TOCC CC (Blue) 

6 

3 


At first glance the table would lead one to believe that the TOC 
Commander had an accurate representation of the battle space. This is an 
artifact due to averaging the Blue and Red force marks. The raw data indicate a 
different story (see Table 7). In fact, there were GRADES that indicated up to 8 
Blue force locations (there were only 6 Blue force entities) and up to 4 Red force 
locations (there were only 3 Red force players). 
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Table 7. TOC Commander Raw Data. 


Blue Red Total 
8 0 8 

5 3 8 

6 2 8 

_6 _ 2 _ 8 _ 

5 4 9 

_5 _ 4 _ ^ 

5 3 8 


B. GROUND TRUTH VS. PERCEIVED TRUTH 

Due to the numerous incorrectly completed GRADES and the different 
times GRADE measures were completed, it was impossible to create a common 
picture of the battlespace at any point in time. The next best solution was to 
determine if there were participants who identified each other at close temporal 
points. This criterion change yielded three participants who had completed 
GRADES within seven minutes of each other (see Table 8). This data indicates 
that within a seven minute period, the Blue force knew the exact location of Red 
2 and Red 3. This battlespace knowledge was better than Red 1 (Red 2 error = 
100m; Red 3 error = 200m) or Red 2 (Red 2 error = 200m; Red 3 error = 200m) 
knowledge of the battlespace. 


Table 8. Raw data near temporal point one. 


Participant 

Time 

GRADE# 


Vehicie 

Error in meters 

SORSE 


9:50 

1 

R2 

0 



9:50 

1 

R3 

0 

Redi 


9:44 

2 

R2 

100 



9:44 

2 

R3 

200 

Red2 


9:43 

2 

R2 

200 



9:43 

2 

R3 

200 


One hour later as seen in table 9, the same three participants completed 
GRADES at nearly the same time. This data set spans a six minute period in 
which the Blue force lost Red 2, detected Red 1 (Red 1 error = 700m) and had 
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more error in Red 3; Red 3 error = 300m). Both Red 1 (Red 2 error = 500m; Red 
3 error = 1300m) and Red 2 (Red 2 error = 300m; Red 3 error = 800m) knew of 
their locations. 


Table 9. Raw data near temporal point two. 


Participant 

Time 

GRADE# 


Vehicie 

Error in meters 

SORSE 


10:50 

4 

R1 

700 



10:50 

4 

R3 

300 

Red 1 


10:44 

5 

R2 

500 



10:44 

5 

R3 

1300 

Red 2 


10:50 

5 

R2 

300 



10:50 

5 

R3 

800 


C. AUDIO DATA 

The audio recordings from the following four participants were transcribed: 
TOC Commander, SORSE, Red 1, and Red 2. 

An overall analysis of the data from Blue forces indicates that Blue 
participants primarily reported on the detections and locations of Red forces and 
Red activities. There were four instances of military grid coordinates on Red 
forces being passed over the Blue radio network and many referrals to land 
marks near the Red forces (e.g., under tree or near ranch house). One 
interesting question from the Buster UAV ground control unit to SORSE ground 
forces was, “Have you reported any of these things yet? I’ve just been sending 
them [pictures].” 

TOC Commander audio logs indicated that he provided very little input to 
SORSE. In fact there is only one instance in the two hour scenario that he 
provides information to SORSE about a vehicle coming down a road. 

SORSE audio logs indicated some confusion coming from the Buster 
ground control unit about what pictures to take. There is one instance at 10:10 
am of Blue 3 passing an incorrect report on a Red vehicle location and its 
activities. 
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An overall analysis of the Red forces voice recordings indicates that Red 
forces primarily reported on their own locations and activities. Red 1 audio logs 
indicated that he could hear the Blue forces radio net. In fact, when Blue forces 
would report detecting a Red player, Red 1 would report in to Red commander 
that Blue forces were correct. Red 1 had the least amount of data out of these 
four analyses. 

Red 2 audio logs indicated occasional UAV sightings, but mainly consist of 
comments on Red force locations and activities. Red 2 was very concerned with 
STARTEX and FINEX times for the scenario. 
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V. DISCUSSION 


Some members of the Department of Defense believe the NCW assertion 
that technology will provide a multiplicative effect to US military forces resulting in 
improved capability and mission effectiveness. However, to date there are no 
empirical findings that support or refute the claims put forth by the architects of 
NCW. 

This thesis proposed an objective method to evaluate NCW system 
assumptions by objectively assessing system performance using a process 
tracing approach. Data collection occurred on 17 May 2007 in the field at Camp 
Roberts, California during TNT 07-03 field experiment. Due to a high data loss, 
statistical analysis for supportable conclusions could not be drawn from the 
GRADE measure. Thus the DMSC could not be validated by this thesis in a field 
setting. 

Following is an analysis of what went well and what went awry with the 
overall GRADE administration including instructions for GRADE, adequacy of 
training for participants filling out GRADE, quality control issues for the GRADE, 
and finally, a discussion on the scope of the experimental data collected. 


A. GRADE DATA 

Many factors contributed to the inability of the GRADE measure to provide 
usable data. An after action review of the researchers’ processes and 
procedures to populate the DMSC and evaluate the GRADE measure provided 
the following items as factors that hampered the data collection effort. 

Oval 1 - Ground Truth was collected by having all vehicles equipped with 
GPS that broadcasted their locations to the TNT network. After an initial review 
of the GPS data it was noted that there were some issues with this technological 
solution establishing ground truth. Multiple vehicles had time periods for which 
no GPS data were recorded; the Raven UAV yielded no usable data. Access to 
GPS data for a fast moving vehicle (e.g., a UAV or automobile) is critical to 

35 



accurately place it in the battlespace over the course of a scenario. The need for 
frequently updated GPS location data for vehicles that are moving slowly or 
standing still is less critical. 

If the DMSC is to be a viable approach to the assessment of NCW it is 
important to consider the issues and challenges associated with establishing 
ground truth. GPS systems were the technological method chosen to record the 
participants’ locations at the TNT event. Although GPS systems are fairly 
accurate, the data they yield still has some degree of error. There is a possibility 
of an error of up to 100 meters from the physical GPS location due to a variety of 
factors (e.g. atmospheric conditions, ephemeris error, clock drift, measurement 
noise, selective availability, multipath, dilution of precision, obstruction). 

Oval 2 - Sensor Data were collected by determining the platform’s 
capability and then by collecting data on where the sensor was actually pointed. 
At the TNT event, researchers had a full understanding of the platform’s 
capability prior to the scenario. In the later case, there are two particular sensor 
issues that need to be resolved.. The first concerns UAVs and the second the 
human participants. 

For this TNT, there were GPS locations for the Buster UAV; however, the 
GPS locations were not available for what the camera saw. This problem 
contributed to errors in scoring the data for detections. For example, it is 
possible the ground truth data will indicate Buster flew over Redi which may be 
could potentially be a hit. Flowever, further analysis may indicate that Buster was 
actually banking hard to the right when it passed over Redi, putting the camera’s 
focus away from the target. In terms of the DMSC model, ground truth or data 
populating Oval 1 never made it to Oval 2 due to the sensors inability to detect 
Redi. This missing data will propagate through the model presenting a false 
representation of the battlespace for Oval 3 which results in the participants’ 
inability to perceive this data for Oval 4 and comprehend it in Oval 5.. 

Another problem occurred when trying to capture where the human 
sensors (i.e., the exercise participants) were actually looking. Currently, there is 
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not a process in place to collect these data during TNT experiments. This 
inability to capture the actual information perceived by the participants leads to 
scoring errors when trying to determine hits, misses, or other categories. As 
discussed in Chapter III, it is not enough to know where a sensor platform is 
located, it is also essential to know where it is looking in order to populate Oval 2. 
This applies to both human and technological sensors. 

Ova/3 -Oval 3 was populated by capturing screen shots of the Air Boss’s 
monitor in the TOC at 15 minute intervals. Fifteen minute data sample intervals 
do not provide enough fidelity to populate Oval 3. Only five of six screen 
captures were potentially useful, and of those, only two aligned temporally with 
two different participants’ GRADE measure completion times. One of the main 
reasons screen captures were not usable was because they did not have time 
stamps on them, making them difficult to integrate them into the other data. 

Oval 4 & 5 - Oval 4 (Perception) and Oval 5 (Comprehension) data were 
collected using the GRADE and digital voice recorders. In the case of the former, 
all participants made errors in indicating their understanding of the battlespace 
on the GRADE maps. On a few GRADES, only one color marker (blue or red) 
was used to identify both Red and Blue forces. Often there were blue or red 
marks but no identification with which to compare that mark to ground truth. In 
other instances, unique identifiers were used but the researchers could not 
determine what they represented. 

There is not a consistent temporal referent across data samples since 
participants did not complete the GRADE at the precise time they were 
instructed. The morning scenario on 17 May 07 had an official start time of 09:04 
with an expected first GRADE of 09:24. Actual initial GRADE start times ranged 
from 09:20 through 09:40. Within a participants GRADE measures there was 
variability such that the GRADES were not always taken at twenty minute 
intervals. In fact, there was enough temporal discrepancy that three participants 
completed one additional GRADE. 
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The digital voice recorder data were not without error either. Some of the 
voice recordings were lost when some participants accidentally turned off the 
digital voice recorders or when participants placed the recorder down and walked 
away, forgetting that they were supposed to keep it on their bodies. Another 
issue with digital voice recorder data is the lack of speech intelligibility due to 
background noise or wind blowing over the microphone. 


B. GRADE INSTRUCTIONS 

The inconsistencies in how each participant filled out their GRADE 
measures indicated that the instructions were not clear. One apparent issue 
with the GRADE instructions is that it was not clearly stated that when marking 
the location of Blue or Red forces, the participant must identify the blue or red 
mark. For example, if the participant knew that the Honda SUV belonged to Red 
1 they should have placed a red dot on the map and annotated near it, “Red 1.” 

Participants also did not follow the first instruction which stated that 
participants needed to look away from their displays while completing the 
GRADE. Two participants’ GRADES closely mirrored ground truth, indicating 
they were using the GRADE to represent what was on their displays (Oval 3 
data) rather than where they were focusing. 


C. GRADE TRAINING 

Due to inconsistencies in how participants filled out their GRADES, there 
appears to be a lack of understanding by the participants of what is required to 
complete the forms. These inconsistencies clearly indicate inadequacy in 
training. This should not be surprising since there were no formal processes or 
procedures for the training of participants in how to complete a GRADE. This 
inadequacy was due to time limitations of participants and the lack of personnel 
available to conduct the training. Again, this is not surprising since participants in 
the TNT experiments do not have much extra time. Typically, participants are so 
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heavily task loaded to execute the scenarios they do not have time for a formal 
class before or between scenarios to receive training. 

The second reason is the lack of personnel available to participate in data 
collection. Generally one or two researchers try to spend a few minutes with 
each participant to offer ‘just in time training’ before a TNT scenario began. For 
the scenario this thesis investigated, there was only one researcher available to 
collect data from nine participants, most of whom were not collocated with the 
researcher. 


D. QUALITY CONTROL FOR GRADE 

There were no formal processes or procedures for quality control checks 
of the GRADES completed by the participants. This is primarily due to the 
assumptions that participants would receive training in how to complete the 
GRADE and that the GRADE was not difficult to complete. As already 
mentioned, it was impossible for a single researcher to check on the nine 
participants’ GRADE measures each time they were completed. 


E. RESEARCH EFFORT SIZE 

The inadequacy of the GRADE data collection resulted from researchers 
trying to capture too much data in the hope of being able to reconstruct ground 
truth for the entire two hour scenario. This goal overstretched the limited 
resources and personnel available for these data collection efforts. All the issues 
discussed up to this point can be traced back to the problems that arose when 
trying to collect too much data with too few assets. 

In the event that this research effort had collected all the data initially 
planned, the total amount of data would have been enormous and would have 
taken several months to process for analysis. For example, processing the data 
for Ovals 4 and 5 on the TOC CDR, SORSE, Red1 and Red2 required 
approximately 40 hours of effort from four full-time research assistants. 
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Completing this analysis of what went well and what went awry with the 
processes and procedures of administering the GRADE measure provided 
researchers with a framework in which recommendations could be generated 
and a future course of action could be planned for the next data collection 
opportunity. 
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VI. CONCLUSION 


A. CONCLUSION 

Network Centric Warfare systems need to be empirically tested in order to 
validate developers’ claims of enhanced capability and mission effectiveness. 
The Dynamic Model of Situated Cognition was selected as a way to model and 
test NCW systems due to its proven merit and recognition by many organizations 
as a way to model human system performance. The goal of this research project 
was to determine if the GRADE could be used as a data collection strategy and 
process by which the DMSC model could be validated in a field setting for NCW 
research. Unfortunately, that goal was not fully realized with the TNT 07-3 data 
collection effort. However, it was not the DMSC model or the GRADE that 
contributed to the poor data collection results, but rather inadequacies in the 
process and procedures employed to populate the DMSC and GRADE. The 
results of this research effort in no way refute the efficacy of the GRADE or the 
DMSC to model NCW in the field. These results have brought to light procedural 
errors in the methods used on GRADE administration. 

Following are recommendations that will optimize GRADE data collection 
in future TNT experiment opportunities. Then, the next section describes the 
need for an automated or digital GRADE, which is then followed by a section 
describing a different strategy to analyze certain events of interest. 


B. RECOMMENDATIONS 

GRADE instructions. Since the TNT exercise occurs every quarter or 
so, participants have had experience in completing GRADES in past TNT 
experiments. Participant errors indicate the instructions have some ambiguity 
and allow for interpretation of intent. Following are suggested instructions to 
improve the quality of GRADES. 
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1. Grade Instructions for All Participants 

Please complete the GRADE every 20 minutes after STARTEX until 
FINEX according to the following instructions. 

1. Do not look at any display while completing GRADE. 

2. Record your duty position (e.g., Air Boss, Red1, etc.) and the 

current local time (using GPS or cell phone) in the upper left 
portion of the map. 

3. Use the blue marker to indicate where you believe the Blue forces 
are located; 

4. Use the blue marker to identify who you think it is. If you are 
unsure of which Blue force it is, put a question mark next to the 
location mark. 

5. Draw a circle around each Blue entity to indicate your 
confidence/certaintv that they are located where you indicated. A 
small circle indicates high confidence/certainty. A large circle 
indicates low confidence/certainty. 

6. Follow the same procedure for Red Forces with your red marker. 

7. Use the red marker to indicate where you believe the Red forces 
are located and then use the red marker to identify who you think 
it is. If you are unsure of which red force it is then use a question 
mark to identify the location mark. Draw a circle around each red 
entity to indicate your confidence/certainty that they are located 
where you indicated. A small circle indicates high confidence/ 
certainty. A large circle indicates low confidence/certainty. 

GRADE training. Training will reduce inconsistencies in how participants 
complete GRADES. Creating a training folder is a solution that should facilitate 
training in as short a time as possible. The training folder could consist of: 
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1. A three ring binder. 

2. Accurate geo-referenced maps of the area of operations. 

3. Opposite the GRADE map should be a page with instructions 
indicating what needs to be done to complete each practice 
GRADE (GRADES should be completed until there are no errors). 

4. A blue and a red permanent marker should be provided. 

The instructions for participants to follow to complete the practice GRADE 
could include the following: 

1. Your position is Red1. 

2. The time is 09:20. 

3. Indicate the placement of Blue2 anywhere on the map and indicate 
that you are highly confident in that location. 

4. Indicate the placement of Red3 anywhere on the map and indicate 
you have low confidence in that location. 

5. Indicate the placement of a Blue force on the map that you cannot 
specifically identify and indicate you are highly confident of its 
location. 

Researchers should generate several versions of the practice instructions 
varying time and Blue/Red force identity. 

Recommended procedures to follow for training are as follows: 

1. Have the participant read the GRADE instructions. Answer any 
questions the participant may have. 

2. Flip to the first GRADE practice problem and have the participant 
read the instructions and then complete the practice GRADE. 

3. Review the Practice GRADE for errors and have the participant 
complete another practice GRADE. Repeat until the participant has 
completed three in a row correctly. 
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Using this method to complete GRADE training should ensure correctly 
completed measures and usable GRADE data returns. 

Quality control checks for GRADE. Quality control checks should occur 
immediately after each practice GRADE and after the first actual GRADE is 
completed. If the suggested instructions and training are used, quality control 
checks may need to be conducted when the participants turn in their completed 
GRADES. 

C. FUTURE FOR GRADE 

One possible solution to reduce data collection error is to create a digital 
or automated GRADE. This will reduce the overhead of supplies needed to 
complete the GRADE. A digital GRADE could also provide the capability of 
forced choice responses thus removing a major contributor to the error in the 
data collected for TNT 07-03. Furthermore, a digital GRADE will ensure 
standardization in how participants complete the GRADE and will provide a 
consistent temporal referent. The proposed computerized GRADE should have 
the following capabilities: 

1. File Manipulation Capabilities 

. Import geographic maps with specific UTM, Lat/Long or MGRS ranges. 

. Save and load the currently annotated geographic map. 

. Export user annotations and the result of disparity analysis to a 
commercially available spreadsheet software (e.g., MS Excel) for 
tabulation and statistical analysis. 

• Import user defined symbols to complement MIL-STD 2525B symbols 
(i.e., common warfighting symbology) and provide them in the 
annotation tool bar. 

2. Annotation Capabilities 

. Capability to indicate position and strength of friendly, enemy and other 
individual and unit icons on the map. 
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. Software library of icons (including MIL-STD 2525B) and the capability 
to create and add new icons. 

. ‘Click and drag-able’ capability for icons. 

. Ability to place confidence or certainty circles around icons. 

. Ability for user voice-over narration while drawing. 

. Ability to draw battlefield graphics (e.g., avenues of approach and axis 
of advance). 

. Ability to collect and distinguish “ground truth” from perceived or 
predicted truth. 

3. Analysis Capabilities 

. Real time analysis of current data. 

. Compute disparity between ground truth and reported conditions. 


D. FUTURE STRATEGIES FOR POPULATING DMSC 

Researchers need to reevaluate the current Ground Truth data collection 
strategy. Researchers are using a process tracing method to examine the 
effects of task, environmental, and individual difference factors over the course of 
a two hour scenario to make inferences about decision strategies and the 
situated cognition that may have contributed to choosing a course of action. To 
accomplish this goal, researchers try to collect as much data as possible to build 
an accurate representation of what occurred during a two hour scenario. This 
large amount of data must be collected from multiple sources and time stamped. 
It is critical to have a single temporal referent when combining multiple data 
sources to achieve an accurate representation of what occurred, more over with 
out a single temporal referent it may become impossible to combine multiple data 
sources. 

One data source that needs to be changed is the current method used to 
populate Oval 3. With the benefit of hindsight, it is apparent that more frequent 
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screen captures would have been better than once every fifteen minutes. A 
better method to collect data for Oval 3 would be to video tape the display of 
interest and store this video recording on a hard drive. Storing the footage on a 
hard drive would remove the step of converting video tape storage to hard drive 
storage for later analysis. 

Rather than trying to reconstruct an entire two hour scenario, it may be 
better to identify key events of interest and then perform process traces on these 
events. Smaller vignettes require less data to populate the DMSC and are easier 
to analyze. Then, over time as data collection processes and procedures are 
refined, researchers can attempt to capture entire scenarios. Researchers 
should collaborate with TNT planners to include smaller or more focused 
vignettes for data collection opportunities. These vignettes would maximize 
researchers’ efforts to collect data and ensure that the right data are collected to 
allow a process trace of events of interest through the DMSC. The vignette data 
collection efforts would be a useful step to demonstrate the efficacy of GRADE 
as a method and process by which the DMSC model can be validated and 
applied for field research of NCW systems. 

The GRADE continues to demonstrate potential to capture situated 
cognition not only in laboratory settings, but also in field settings. Following the 
recommendations in this thesis it will be possible to capture Ovals 4 and 5 in 
future research. This provides the ability to populate all the Ovals of the DMSC 
model. The DMSC then could be used to empirically test different NCW 
systems, thus providing decision-makers a more thorough understanding of the 
capabilities of each NCW systems. 
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APPENDIX 


GRADE 


Instructions for GRADE 

(Geographical Recall and ^n^lysis of Data in the Environment) 


Please complete the GRADE every 20 minutes after STARTEX until FINEX according to the 
following instructions. 

1. Turn away from your information display, monitor, etc. 

2. Record your duty position (e g.. Air Boss, Red 1, etc.) and the current time in the upper left 
portion of the map. 

3. Use the blue marker fo identify where you believe the blue forces are located. Draw a 
circle around each blue entity to indicate your confidence/certainty. A small circle Indicates 
high confidence/certainty. A large circle indicates low confidence/certainty. 

4. Use the red marker to identify where you believe the red forces are located. Draw a circle 
around each red entity to indicate your contidence/certainty. A small circle indicates high 
confidence/certainty. A large circle indicates low confidence/certainty. 
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